Most computer search engines respond to user queries by generating a list of documents (the “query documents”) deemed relevant to the query. Document relevancy is determined by measuring the similarity of the retrieved query documents to the query, usually by estimating the similarity between the words in the document and the keywords of the query. The “keywords” usually are the words of the query excluding prepositions, articles, etc.
FIG. 1 shows a search query 10, for “workplace situation awareness” and a list of 10 query documents 12. Each query document 12 is listed by its title, a date for the document and the relevancy of the document to search query 10. Moreover, query documents 12 are presented in the order of relevancy; the first document is the one with what the search engine determines is the highest relevancy (80%) and the remaining documents are less relevant according to the search engine.
Unfortunately, the search engine's assumption of relevancy is not always correct and the user may determine that few of the top 10 query documents are truly relevant. One measure of this is called “precision at 10” or “P@10” and is the number of documents in the top 10 results that are truly relevant to the query. The P@10 is manually generated by having a specialist on the subject of the query read each document in the collection and mark it as relevant or irrelevant to answering the query.
Another measure of the quality of the search results is the mean average precision (MAP). This is a metric that measures how high the relevant documents were ranked compared to the irrelevant ones. There are several methods to compute the average precision, but all essentially measure the area under a precision versus recall curve, where precision is the precision at N (N varying between 1 and a large number, for example 1000) and recall is the number of documents ranked as better than the Nth relevant document. The book Modern Information Retrieval by Ricardo Baeza-Yates and Berthier Ribeiro-Neto, Addision-Wesley, 1999, discusses many of these issues.
Most search engines have difficulty answering certain queries. For example, consider the query “What impact has the Chunnel had on the British economy and/or the life style of the British?” Most search engines will return many irrelevant documents, containing the words ‘British’, ‘life’, ‘style’, ‘economy’, etc. But the gist of the query, the Chunnel, is usually lost.
There has been a movement to predict the quality of a search results. The following two articles discuss query prediction.
Cronen-Townsend, S., Zhou, Y. and Croft, W. B., .“Predicting Query Performance,” Proceedings of SIGIR 2002, Tampere, Finland, Aug. 11-15, 2002, pp. 299-306.
Giambattista Amati, Claudjo Carpineto, and Giovanni Romano “Query Difficulty, Robustness and Selective Application of Query Expansion”, Advances in Information Retrieval, 26th European Conference on IR Research, ECIR 2004, Sunderland, UK, Apr. 5-7, 2004, pp 127-137.
Unfortunately, these articles discuss methods which are tied to particular search engines and thus, are not easily transferable to another search engine.
It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements.